In recent years, mobile telephones have increasingly become equipped with built-in camera modules. These camera modules enable a user to take a digital photograph and quickly transmit it to friends and family.
A generic digital camera system that is part of a mobile telephone or other electronic device is shown at 10 in FIG. 1. The digital camera 10 includes a housing 11 which contains at least one lens 12 and at least one image sensor 18 which is operatively connected to a memory unit 14 and a processor or controller 16, both of which are located outside of the camera module 10 and inside of the mobile telephone or other electronic device within which the camera module 10 is housed. The memory unit 14 can be used to store digital images and computer software for performing various functions in the digital camera 10, as well as to implement the present invention. The image sensor 18 can be a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or another system.
Camera modules that are used in mobile telephones are commonly designed to be both small and inexpensive for many reasons. As a consequence of this fact, a typical image sensor 18 does not include any frame memory within the camera module 10. Instead, the image data from a captured image must be output to the telephone engine as soon as it is read from the sensor. Otherwise, the image data would be lost. The image sensor 18 is typically read in a line- by-line fashion, and the data is sent in the same order to the telephone engine, where there exists sufficient memory for storing the data.
In order to reduce the bandwidth needed for transmission of the image data between the camera module 10 and telephone engine, the image sensor 18 often compresses the data before sending it any further. In order to reduce the processing requirements in the telephone engine, many camera modules use JPEG compression at this stage, as the images would likely end up in a JPEG format in any case so that the telephone engine does not need to consume additional time for JPEG encoding.
From a usability perspective, it is not sufficient for the camera images to only be processed and saved to memory. This is because the end-user would also like to view the image he or she has just taken on the screen of the telephone, for example, to decide whether or not the image was of a satisfactory quality. It is also desirable from the user's perspective for there not be any additional delay in showing the preview image to the user.
Unfortunately, however, making a preview image out of the compressed data is not an immediate operation, but instead inevitably takes some additional time. There are a number of different potential solutions to this issue, but each possesses serious drawbacks of its own. One such solution involves allowing the camera module 10 to output the data of the captured image in such a format to allow the quick construction of the smaller preview image that is to be shown on the telephone display. In practice, this means using an uncompressed format for the image data. The problem in this approach, however, is that with higher resolution cameras, there might not be enough bandwidth to transfer the data.
Another potential solution to this problem is to use a camera module that includes sufficient frame memory to permit the same image to be transmitted twice, first as an image with proper size for the display, and then a full resolution image for saving purposes. The problem with this approach is that additional memory is simply too expensive to practically implement this solution.
Another potential solution to the problem discussed above involves interleaving a compressed image for saving and an uncompressed preview image for viewing. Even though there is no memory for a whole image frame in the camera module, a JPEG encoder in the camera requires that there be memory for 8 lines of the image. This makes it possible to interleave the uncompressed preview image data within the compressed full resolution image data. Because the preview data is intended for the mobile telephone's display (whose resolution is substantially lower than the one of the camera sensors), the interleaving of the downscaled preview image does not increase the bandwidth requirements by a large amount. The process of interleaving, where two types of data are split into chunks and sent one after another, is depicted in FIG. 2. In FIG. 2, the white and black diamonds represent specific markers that mark the beginning of each type of chunk. In practice, the markers must be chosen so that they cannot appear in the data itself. Alternatively, one could include information about the length of the chunk into each marker. However, in JPEG encoding, for example, it is generally not possible to know the size of the output data before the encoding has taken place. The camera should therefore be equipped with additional buffers for this approach.
Transmitting all of the data of one image frame requires the transmission of several chunks of preview data and compressed data. One chunk can contain as much information as the camera module is able to buffer at any given time. For example, if a JPEG encoder requires that 8 lines must be available in a buffer of some sort, then it is possible to use chunks that contain data at least for those 8 lines. It should be noted that there might be some additional buffering capability as well in the camera module.
FIG. 2 also illustrates how chunks of preview image data always have the same length, while the length of compressed data chunks may vary. This is because the compression efficiency depends upon the image content and can vary even within one single image. The preview data, on the other hand, is uncompressed and there is therefore always the same amount of preview data to be sent in one chunk. For this reason, the markers after the preview data chunks (at the beginning of the compressed data chunks) are not obligatory, as it is the length of the preview data chunk is always known.
Although the interleaving of compressed and uncompressed data addresses many of the issues that arise in the other potential solutions discussed previously, it still has a significant deficiency. With this approach, even though compressed data is available for storing purposes and preview data available for displaying purposes, the interleaved data must first be deinterleaved before it can be used. In other words, the compressed data must be extracted from the preview data, or the preview data must be extracted from compressed data, before the compressed data can be saved and the preview data can be viewed.
In most conventional mobile telephones with a built-in camera module, once all of the image data has been received into a memory inside the telephone engine, it is processed or copied further by the phone processor in such a way that all of the data contained within the image is addressed in turn. In the case of interleaved data, the data is copied into one of the two buffers upon processing. One buffer is dedicated for storing the preview (uncompressed) data, and the other buffer is dedicated for storing the compressed data. It should be noted that in practice, one of these two buffers may be the same as the original buffer, in case the original interleaved data is no longer needed but can be overwritten. However, this process takes consumes both time and processing power.
It would therefore be desirable to develop a system that enables the more rapid and efficient transmission and availability of both compressed and uncompressed image data in a mobile telephone or other electronic device.